Tools

A new computer program can accurately predict how our brains respond to any noun, whether "celery", "airplane", or "kumquat".

The program guesses a word's meaning from its occurrence in a huge volume of internet text, and then builds a mental picture of the word based on the brain's reaction to other words.

"I think of it as us beginning to break the brain's code," says Marcel Just, a neuroscientist at Carnegie Mellon University in Pittsburgh, US, who led the study along with colleague Tom Mitchell.

Their approach was only made possible by access to a Google database of one trillion words – the equivalent of 1.8 million copies of Tolstoy's War and Peace. The company released the dataset to help computer programs predict the meaning of sentences and stories.

Verb neighbours

Instead, Mitchell and Just calculated how often each concrete noun – something that can be touched, smelled, seen, etc – occurs next to 25 common verbs, such as "see", "hear", "move" and "taste."

The 25 verbs are fundamental enough to capture the meaning of any concrete noun, Mitchell says. "Telephones are what you do with them and you don't eat them and you do listen to and speak to them."

The team next scanned the brains of nine college students as they viewed 60 words alongside a picture of each word.

They used a functional MRI (fMRI) scanner, which measures activity of the working brain via changes in blood flow. The words belonged to 12 categories, such as animals, vegetables and vehicles.

Based on these 60 brain scans and the trillion-word library, Mitchell and Just created a computer program for each volunteer that predicts how their brain will respond to any concrete noun.

Beating the odds

"We don't want to do fMRI studies of all 50,000 words in English, then move on to other languages. We want to understand the principle by which the brain represents things," Just says.

As a first test, the computer program was challenged to predict brain scans of two out of the 60 test words, based on brain images from the other 58 – a 50/50 proposition.

The program beat the odds and matched predicted brain scans to the noun 77% of the time.

Next the program had to predict the correct scan from one word out of 1000 new words. It ranked all the words in order from best match to worst, and averaged over nine people, placed the correct brain image in the top 300, far exceeding chance.

The researchers could only confirm the model's accuracy for words they had gathered brain scans for, but other nouns should be no different, Just says. "I think if you did 'kumquat' and 'balloon', it really should work."

'Love' response

John Dylan Haynes, a neuroscientist at Bernstein Center for Computational Neuroscience in Berlin, says the approach sidesteps one challenge of brain reading:

"The brute force way is to measure the pattern of brain activity associated with each thought one wants to read out," he says. But using this linguistic shortcut, you can deduce the meaning of a word based on it relation to a known verb.

Extracting a word's meaning from its relation to verbs also offers insight into how our brain represents objects, says Sean Polyn, a neuroscientist at the University of Pennsylvania in Philadelphia. "I think it says a lot about the way concepts are built up out of experience."

Brain areas involved in taste, for instance, tended to light up when subjects viewed the word "apple". Less clear is how the brain responds to abstract nouns, Mitchell says. The team is currently testing whether it can predict how the brain will respond to nouns like "love" and "democracy". Adjectives and even phrases will present even more of a challenge.

"If I said 'the fast rabbit' versus 'the hungry rabbit', you think two different meanings," says Mitchell. "I'd love to understand how adding an adjective modifies the neural representation of rabbit."

Eventually the hope is that the method will help understand the patterns in the brain, and how thoughts are constructed. Decoding how the brain works will probably lead to better artificial intelligence, says Just. "If it works well in the brain maybe it will work well in silicon."

Journal reference: Science (DOI: 10.1126/science.115876)